Fair and Open Evaluation May Call for Temporarily Hidden Authorship, Caution When Counting the Votes, and Transparency of the Full Pre-publication Procedure

The problem of imbalance The traditional system of manuscript evaluation has created a certain imbalance bordering with unfairness: while the authors of submitted papers typically have had their identity disclosed already at the outset, the reviewers have remained mostly anonymous. With a new open evaluation system being currently envisaged, the main difference would be that evaluators become disclosed as well, which is a significant step toward balance and fairness. More openness and constructive interactivity in the reviewing process have become to be practiced increasingly more, including some noteworthy success (e.g., the Shepherding system at the European Conference on Pattern Languages of Programming and Computing and the Frontiers initiative). However, while both the authors and the reviewers have become disclosed, the past collective “instincts” and traditions of reviewers as evaluators could often remain unaffected by this change. Thus, if the new envisaged evaluation system fully discloses both the authors and evaluators, it nevertheless cannot tackle all potential sources of bias and unfairness. Although for the majority of researchers this system seems to be suitable, there are also researchers who feel that not all possible sources of unfairness would be eliminated. Therefore there should be also an additional optional format of review and publishing that goes even further in pursuit for minimizing the impact of subjectivity. Why so?